Small Sample Statistics for Classi cation Error Rates I : Error Rate Measurements

Several methods (independent subsamples, leave-one-out, cross-validation, and bootstrapping) have been proposed for estimating the error rates of classiers. The rationale behind the various estimators and the causes of the sometimes con BLOCKINicting claims regarding their bias and precision are explored in this paper. The biases and variances of each of the estimators are examined empirically. Cross-validation, 10-fold or greater, seems to be the best approach; the other methods are biased, have poorer precision, or are inconsistent. Though unbiased for linear discriminant classiers, the 632b bootstrap estimator is biased for nearest neighbors classiers, more so for single nearest neighbor than for three nearest neighbors. The 632b estimator is also biased for Cart-style decision trees. Weiss' loo* estimator is unbiased and has better precision than cross-validation for discriminant and nearest neighbors classiers, but its lack of bias and improved precision for those classiers do not carry over to decision trees for nominal attributes.